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Abstract 

in this paper, we present our efforts towards developing an Intelligent tourist system. Th 
with a unique combination of sensors and software. The hardware Includes two comput« 
lapel microphone plus an earphone, a video camera and a head-mounted display. This 
multimodal interface to take advantage of speech and gesture input to provide assistant 
software supports natural language processing, speech recognition, machine translatior 
recognition and multimodal fusion. A vision module is trained to locate and read written 
adapt to to new environments, and is able to interpret Intentions offered by the user sue 
clarification or pointing gesture. We illustrate the applications of the system using two e) 
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^ Posters & demos: An automatic sign recognition and translation system 
Jie Yang, Jiang Gao, Ying Zhang, Xilin Chen, Alex Walbel 

November 2001 Proceedings of the 2001 workshop on Perceptive user interfaces PUI 
•01 

Pubiisher: ACM Press 

Full text available: Qpdf(1.21 MB) Additional Information: full citation , abstract , references 

A sign is something that suggests the presence of a fact, condition, or quality. Signs are 
everywhere in our lives. They make our lives easier when we are familiar with them. But 
sometimes they pose problems. For example, a tourist might not be able to understand 
signs in a foreign country. This paper discusses problems of automatic sign recognition 
and translation. We present a system capable of capturing images, detecting and 
recognizing signs, and translating them into a target language. We d ... 
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Pattern recognition and synthesis for sign language translation system 
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October 1994 Proceedings of the first annual ACM conference on Assistive 
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Full text available: fi!lDd«721.82 KB) Additional Information: full citation , abstract , references , citings , index 
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Sign language is one means of communication for hearing-impaired people. Words and 
sentences in sign language are mainly represented by hands' gestures. In this report, we 
show a sign language translation system which we are developing. The system translates 
Japanese sign language into Japanese and vice versa. In this system, hand shape and 
position data are inputted using DataGlove. Inputted hand motions are recognized and 
translated into Japanese sentences. Japanese text is translated int ... 
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Full text available: ^ p df (208.1Q KB) Additional Information: full citation , abstract , references , citings 

Signs are everywhere in our lives. They make our lives easier when we are familiar with 
them. But sometimes they also pose problems. For example, a tourist might not be able 
to understand signs in a foreign country. In this paper, we present our efforts towards 
automatic sign translation. We discuss methods for automatic sign detection. We describe 
sign translation using example based machine translation technology. We use a user- 
centered approach In developing an automatic sign translation syst ... 

Keywords: sign, sign detection, sign recognition, sign translation 
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Volume 6 Issue 1 
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This article presents a transfer-based statistical model for Chinese to Taiwanese sign- 
language (TSL) translation. Two sets of probabilistic context-free grammars (PCFGs) are 
derived from a Chinese Treebanl< and a bilingual parallel corpus. In this approach, a 
three-stage translation model is proposed. First, the input Chinese sentence Is parsed into 
possible phrase structure trees (PSTs) based on the Chinese PCFGs. Second, the Chinese 
PSTs are then transferred Into TSL PSTs according to the t ... 
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Posters: Gesture-driven American sign language phraselator 
Jose L'Hernandez-Rebollar 

October 2005 Proceedings of the 7th international conference on Multimodal 
interfaces ICMI '05 

Publisher: ACM Press 

Full text available: Q pdf(306.63 KB) Additional Information: full citation, abstract , references, index terms 

This paper describes a portable American Sign Language (ASL)-to-English phraselator. 
This wearable device is based on an Acceleglove originally developed for recognizing the 
hand alphabet, and a two-link arm skeleton that detects hand location and movement 
with respect to the body. Therefore, this phraselator is able to recognize finger-spelled 
words as well as hand gestures and translate them into spoken voice through a speech 
synthesizer. To speed-up the recognition process, a simple predictio ... 

Keywords: ASL translation, gestural interfaces, gesture recognition 
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7 Design challenges: American sign language recognition in game development for 
^ deaf children 

Helena Brashear, Valerie Henderson, Kwang-Hyun Park, Harley Hamilton, Seungyon Lee, 
Thad Stamer 

October 2006 Proceedings of the 8th international ACM SIGACCESS conference on 
Computers and accessibility Assets '06 

Publisher: ACM Press 

Full text available: ^pdf(341.40 KB) Additional Information: full citation , abstract , references , index terms 

CopyCat Is an American Sign Language (ASL) game, which uses gesture recognition 
technology to help young deaf children practice ASL skills. We describe a brief history of 
the game, an overview of recent user studies, and the results of recent work on the 
problem of continuous, user-independent sign language recognition in classroom settings 
Our database of signing samples was collected from user studies of deaf children playing 
aWizard of Oz version of the game at the Atlanta Area School for th ... 

Keywords: ASL, game, recognition, sign language 



Session 5: novel interaction: A teaching system of Japanese sign language using 
sig n language reco g nition and generation 
Hirohiko Sagawa, Maseru Takeuchi 
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Publisher: ACM Press 

Full text available- 155 Ddf(31 2 40 KB) Additional Information: full citation , abstract , references , citings , index 
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In recent years, the number of sign language learners is increasing in Japan. And there 
are many teaching materials of sign language such as textbooks, videotapes and software 
for PCs. However, these teaching materials have several problems that learners cannot 
study sign language sufficiently because the learners can mainly study manual gestures, 
cannot change the direction to see signed gestures, and cannot checl< their signed 
gestures by themselves. We developed a sign language teaching system ... 

Keywords: 3DCG animation and teaching system, recognition, sign language 
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^ November 2000 Proceedings on the 2000 conference on Universal Usability CUU 'GO 
Publisher: ACM Press 

Full text available: gDdf(471.06 KB^ Additional Information: full citation, abstract , references , citings, index 



http://portal.acm.org/resultsxfin?coll=ACM&dl=ACM&CFm=21 147135 4/27/07 



Results (page 1): sign recognition and translation 



Page 4 of 7 



terms 

An information kiosk with a JSL (Japanese sign language) recognition system ttiat allows 
hearing-impaired people to easily search for various kinds of Information and services was 
tested in a government office. This kiosk system was favorably received by most users. 



Keywords: hearing impaired, information kiosk, recognition, sign language 



11 A unified approach in speech-to-speech translation: integrating features of speech 
recognition and machine translation 

Rulqiang Zhang, Genichiro Kikui, Hirofumi Yamamoto, Taro Watanabe, Frank Soong, Wai Kit 
Lo 

August 2004 Proceedings of the 20th international conference on Computational 
Linguistics COLING '04 

Publisher: Association for Computational Linguistics 

Full text available: 'g pdf (128.32 KB) Additional Information: full citation , abstract, references 

Based upon a statistically trained speech translation system, in this study, we try to 
combine distinctive features derived from the two modules: speech recognition and 
statistical machine translation, in a loglinear model. The translation hypotheses are then 
rescored and translation performance Is Improved. The standard translation evaluation 
metrics, including BLEU, NIST, multiple reference word error rate and its position 
independent counterpart, were optimized to solve the weights of the fea ... 

12 Development of an American Si gn Lan guag e game for deaf children 
Valerie Henderson, Seungyon Lee, Helene Brashear, Harley Hamilton, Thad Starner, Steven 
Hamilton 

June 2005 Proceeding of the 2005 conference on Interaction design and children IDC 
'05 

Publisher: ACM Press 

Full text available: pdf(478.87 KB) Additional Information: full citation , abstract , references, index terms 

We present a design for an interactive American Sign Language game geared for 
language development for deaf children. In addition to work on game design, we show 
how Wizard of Oz techniques can be used to facilitate our work on ASL recognition. We 
report on two Wizard of Oz studies which demonstrate our technique and maximize our 
Iterative design process. We also detail specific implications to the design raised from 
working with deaf children and possible solutions. 

Keywords: ASL, Wizard of Oz method, children, computer games, deaf, gesture 
recognition, language acquisition 
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July 2002 Proceedings of the fifth internationai ACM conference on Assistive 

technologies Assets *02 
Publisher: ACM Press 

Full text available: ^ pdf(833,57 KB) Additional Information: full citation , abstract, references . Index terms 

TESSA Is an experimental system that alms to aid transactions between a deaf person 
and a clerk in a Post Office by translating the clerk's speech to sign language. A speech 
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recogniser recognises speech from the clerk and the system then synthesizes the 
appropriate sequence of signs in British Sign language (BSL) using a specially-developed 
avatar. By using a phrase lookup approach to language translation, which is appropriate 
for the highly constrained discourse in a Post Office, we were able t ... 

Keywords: Aids for the Deaf, avatars, Interactive systems, speech recognition, 
translation systems 



14 The VI framework program in Europe: some thoughts about speech to speech | 
translation research 

Gianni Lazzari 

July 2002 Proceedings of the ACL-02 workshop on Speech-to-speech translation: 

algorithms and systems - Volume 7 
Publisher: Association for Computational Linguistics 

Full text available: Qpdfd 75.47 KB) Additional Information: full citation , abstract , references , citings 

Significant progress has been made in the field of human language technologies. Various 
tasks like continuous speech recognition for large vocabulary, speaker and language 
identification, spoken information inquiry, information extraction and cross-language 
retrieval in restricted domains are today feasible and different prototypes and systems are 
running. The spoken translation problem on the other hand is still. a significant challenge: 
"Good text translation was hard enough to pull off. Speech ... 

15 Project notes and demos: MT and topic-based techniques to enhance speech | 
recognition systems for professional translators 

Yevgeny Ludovik, Ron Zacharski 

July 2000 Proceedings of the 18th conference on Computational linguistics - Volume 
2 

Publisher: Association for Computational Linguistics 

Full text available: Ddf(387.69 KB) Additional Information: full citation , abstract , references 

Our principle objective was to reduce the error rate of speech recognition systems used 
by professional translators. Our work concentrated on Spanish-to-English translation. In a 
baseline study we estimated the error rate of an off-the-shelf recognizer to be 9.98%. In 
this paper we describe two independent methods of improving speech recognizers: a 
machine translation (MT) method and a topic-based one. An evaluation of the MT method 
suggests that the vocabulary used for recognition cannot be com 

16 S plittin g long or ill-formed input for robust spoken-language translation | 

Osamu Furuse, Setsuo Yamada, Kazuhide Yamamoto 

August 1998 Proceedings of tlie 36th annual meeting on Association for 

Computational Linguistics - Volume 1 , Proceedings of the 17th 
international conference on Computational linguistics - Volume 1 

Publisher: Association for Computational Linguistics 

Full text available: pdf(639.58 KB) 

S Additional Information: full citation , abstract , references , citings 

Publisher Site 

This paper proposes an Input-splitting method for translating spol<en-language which 
includes many long or ill-formed expressions. The proposed method splits input into well- 
balanced translation units based on a semantic distance calculation. The splitting is 
performed during left-to-right parsing, and does not degrade translation efficiency. The 
complete translation result is formed, by concatenating the partial translation results of 
each split unit. The proposed method can be incorporated into ... 
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Chung-Chian Hsu, Chien-Hslng Chen, Tien-Teng Shih, Chun-Kai Chen 

April 2007 ACM Transactions on Asian Language Information Processing (TALIP), 

Volume 6 Issue 1 
Publisher: ACM Press 

Full text available: gpdf(1.43 MB) Additional Information: full citation , abstract , references , index terms 

When editors of newspapers and magazines translate proper nouns from foreign 
languages into Chinese, the Chinese translation (termed transliterations) they choose will 
typically be phonetically similar to the original word. With many different translators 
working without a common standard, there may be many different Chinese 
transliterations for the same proper noun, such as using the same sounds but different 
Chinese characters or even using different sounds and characters. This causes confu ... 

Keywords: Chinese transliteration, Romanization, cross-lingual Information retrieval, 
grapheme, phoneme, pinyin, similarity comparison, speech signal processing 
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Multimodal architectures and frameworks: Georgia tech gesture toolkit: supporting 
experiments in gesture recognition 
^ Tracy Westeyn, Helene Brashear, Amin Atrash, Thad Starner 

November 2003 Proceedings of the 5th international conference on Multimodal 

interfaces ICI4I '03 
Publisher: ACM Press 

Full text available: ffi pdf(359.38 KB) Additional Information: full citation , abstract, references , cjtiogs, index 
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Gesture recognition is becoming a more common interaction tool in the fields of 
ubiquitous and wearable computing. Designing a system to perform gesture recognition, 
however, can be a cumbersome task. Hidden i^arkov models (HMMs), a pattern 
recognition technique commonly used in speech recognition, can be used for recognizing 
certain classes of gestures. Existing HI^M toolkits for speech recognition can be adapted 
to perform gesture recognition, but doing so requires significant knowledge of the s ... 

Keywords: american sign language, context recognition, gesture recognition, hidden 
Markov models. Interfaces, toolkit, wearable computers 



20 Learning style translation for the lines of a drawing 
William T. Freeman, Joshua B. Tenenbaum, Egon C. Pasztor 
January 2003 ACM Transactions on Graphics (TOG), volume 22 issue 1 
Publisher: ACM Press 

Full text available- IB Ddf(373 30 KB) Additional Information: full citation , abstract, references , citings , index 
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We present an example-based method for translating line drawings Into different styles. 
We fit each line as a linear combination of similar lines In a training set, and Interpolate 
between the corresponding training examples in the output style. The synthesized lines 
preserve the desired stylistic features of the output style. 
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